Repository navigation
ci: build validation datasets with three speakers - #118
Merged
Merged
Conversation
The training readiness contract requires a speaker-disjoint train/dev/test split with minimum per-class support in every partition. With the default split policy (test 0.25, dev 0.10, minimum 1 per class) GroupShuffleSplit needs three speakers: one held out for test, leaving two so the dev cut can still group by speaker. The builder default of two actors left the dev split empty, so every hosted lane failed readiness before training. Workflows now pass --actors explicitly; the builder default is left alone. A regression test builds each lane dataset and runs the real readiness contract.
Owner
Author
|
Lane results on this branch (dispatched with workflow defaults):
Controls on |
jsugg
added a commit
that referenced
this pull request
Nov 10, 2026
ci: build validation datasets with three speakers
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every hosted validation lane (darwin-x86_64, macos15-mps, linux-3.13-cli, full-dataset quality gate) has failed on
mainsince the training readiness contract landed in #85, attraining_readiness_insufficient_class_supportbefore training starts. Control runs onmain: 42761198900 (quality gate), 42761250998 (linux-3.13).Root cause
scripts/build_synthetic_ravdess_dataset.pydefaults to--actors 1 2. Readiness (_split_findingsinser/_internal/models/training_readiness.py) requires a speaker-disjoint train/dev/test split (split_utterances_three_way) withmin_remaining_per_class_splitsupport per class in every partition.Actor-count derivation
Each synthetic actor records all 8 emotions once, so a class's support in a partition equals that partition's actor count. Default policy (
schema.py):test_size=0.25,dev_size=0.10,min_remaining_per_class_split=1; the quality gate setsSER_TEST_SIZE=0.25, the same value.n_test = ceil(0.25 N), leavingM = N - n_test.0.10 / 0.75:n_dev = ceil(0.133 M); grouping needs at least 2 speakers (dataset_splitting.py,can_group_by_speaker).Empirically, the real readiness check on builder output gives 3 blocking findings for 2 actors and 0 for 3 or 4 actors. The minimum is 3.
Change
--actors 1 2 3explicitly. The builder default is unchanged: no test or doc invokes the builder, and the onlyactors: 2fixtures are unrelated transcription-coverage data.tests/suites/integration/models/test_synthetic_validation_dataset_readiness.py: parses every workflow's builder arguments, builds each dataset, and runs the realrun_training_readinessunder the default split policy. Against the previous workflows it fails with the same findings CI reported.No readiness thresholds or checks were changed.
Validation
Local gates:
make fmt lint type prepush-check ci-contracts test-cov(1209 passed, 79.98%). Lanes dispatched on this branch: results are posted below.